Backlight modulation for liquid crystal display with eyetracking for virtual reality

ABSTRACT

A head mounted display system includes a display device and an eyetracking device. The display device includes a liquid crystal (LC) panel comprising a plurality of rows of pixels, a back light unit (BLU), and a data driver. The BLU emits light during an illumination period of a frame period from an illumination start time and does not emit light for a remaining portion of the frame period. The eyetracking device determines an eye gaze area of a user in a pixel area of the display device. The illumination start time varies based on a location of the eye gaze area of the user. Liquid crystal material in a row of pixels of the LC panel outside the eye gaze area of the user transitions during the illumination period.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of U.S. Provisional PatentApplication No. 62/325,947 filed on Apr. 21, 2016 which is incorporatedherein by reference for all purposes as if fully set forth herein.

BACKGROUND

The present disclosure generally relates to enhancing a Liquid CrystalDisplay (LCD) for use in a virtual reality, mixed reality, or augmentedreality system.

SUMMARY

A head mounted display system includes a display device and aneyetracking device. The display device includes a liquid crystal (LC)panel including a plurality of rows of pixels, a back light unit (BLU),a data driver, and a controller. The controller applies a first controlsignal to the data driver to write data to the LC panel and a secondcontrol signal to the BLU to emit light during an illumination period ofa frame period from an illumination start time and to not emit light fora remaining portion of the frame period. The eyetracking devicedetermines an eye gaze area of a user in a pixel area of the displaydevice. The illumination start time varies based on a location of theeye gaze area.

Also described is a method of displaying an image by a display device.The method includes writing, by a data driver, data to a plurality ofrows of pixels of a liquid crystal (LC) panel, determining, by aneyetracking device, an eye gaze area of a user, and emitting, by a backlight, light at an illumination start time for an illumination periodand not emitting light for a portion of the frame period, wherein theillumination start time is based on the eye gaze area of the user.

In one embodiment, the liquid crystal material in a row of pixels of theLC panel outside the eye gaze area of the user transitions during theillumination period. In an aspect, the eye gaze area of the user is in atop portion of the pixel area and the illumination start time occursbefore an end of the frame period. In another aspect, the eye gaze areaof the user is in a middle portion of the pixel area of the displaydevice, and the illumination start time occurs before an end of theframe period and the illumination period extends into a subsequent frameperiod subsequent to the frame period. Responsive to the eye gaze areaof the user moving below a previous eye gaze area of the user, theillumination start time shifts closer to an end of the frame period.Responsive to the eye gaze area of the user moving above a previous eyegaze area of the user, the illumination start time shifts closer to astart of the frame period. In an aspect, the eye gaze area of the useris in a bottom portion of the pixel area of the display device, and theillumination start time is at an end of the frame period or beginning ofa subsequent frame period. The data driver may write data to a row ofthe pixels of the LC panel outside the eye gaze area in a subsequentframe period during at least a portion of the illumination period. Thedata driver may write data to the plurality of rows of pixels at abeginning portion of the frame period and the illumination start timeoccurs at or after an end of the beginning portion of the frame period.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment including a virtualreality system, in accordance with an embodiment.

FIG. 2A is a diagram of a virtual reality headset, in accordance with anembodiment.

FIG. 2B is a cross section of a front rigid body of the VR headset inFIG. 2A, in accordance with an embodiment.

FIG. 3A is a top view of an example electronic display, in accordancewith an embodiment.

FIG. 3B is a cross section of an example electronic display, inaccordance with an embodiment.

FIGS. 4A, 4B, and 4C are diagrams illustrating a frame cycle of an LCDin black duty insertion mode, in accordance with an embodiment.

FIG. 5 shows a cross section of a headset including a camera fortracking eye position, in accordance with an embodiment.

FIGS. 6A, 6B, and 6C show diagrams illustrating illumination of abacklight unit (BLU) for displaying image data on an LCD and theircorresponding timing waveforms, in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

System Overview

A VR system using an LCD includes an eye tracking apparatus to determinea user's gaze, which allows the system to modulate the illumination timeperiod of the backlight unit (BLU). In a black duty insertion (BDI)mode, image data is loaded to the pixels of the LCD using regularrolling mode from top to bottom over the whole frame time and the BLU isilluminated for only a portion of the frame time (e.g., 20%), forexample at the end of the frame time, to display the entire pixels ofthe display. BLU modulation allows for varying the BLU illumination timeperiod such that the liquid crystals (LCs) corresponding to the pixelsthe user is currently looking at have completed their transition totheir final state before the illumination actually begins. While somepixels of the LCD that is operated in the BDI mode will likely haveeither compromised states or display old image data, BLU modulationenables to confine such compromised pixels within the user's peripheralvision and thereby minimize any degradation of user experience.

FIG. 1 is a block diagram of a virtual reality (VR) system environment100 in which a VR console 110 operates. The system environment 100 shownby FIG. 1 comprises a VR headset 105, an imaging device 135, and a VRinput interface 140 that are each coupled to the VR console 110. WhileFIG. 1 shows an example system 100 including one VR headset 105, oneimaging device 135, and one VR input interface 140, in other embodimentsany number of these components may be included in the system 100. Forexample, there may be multiple VR headsets 105 each having an associatedVR input interface 140 and being monitored by one or more imagingdevices 135, with each VR headset 105, VR input interface 140, andimaging devices 135 communicating with the VR console 110. Inalternative configurations, different and/or additional components maybe included in the system environment 100.

The VR headset 105 is a head-mounted display that presents media to auser. Examples of media presented by the VR head set include one or moreimages, video, audio, or some combination thereof. In some embodiments,audio is presented via an external device (e.g., speakers and/orheadphones) that receives audio information from the VR headset 105, theVR console 110, or both, and presents audio data based on the audioinformation. An embodiment of the VR headset 105 is further describedbelow in conjunction with FIGS. 2A and 2B. The VR headset 105 maycomprise one or more rigid bodies, which may be rigidly or non-rigidlycoupled to each other together. A rigid coupling between rigid bodiescauses the coupled rigid bodies to act as a single rigid entity. Incontrast, a non-rigid coupling between rigid bodies allows the rigidbodies to move relative to each other.

The VR headset 105 includes an electronic display 115, an optics block118, one or more locators 120, one or more position sensors 125, and aninertial measurement unit (IMU) 130. The electronic display 115 displaysimages to the user in accordance with data received from the VR console110. In various embodiments, the electronic display 115 may comprise asingle electronic display or multiple electronic displays (e.g., anelectronic display for each eye of a user).

An electronic display 115 may be a liquid crystal display (LCD), anorganic light emitting diode (OLED) display, an active-matrix organiclight-emitting diode display (AMOLED), a TOLED, some other display, orsome combination thereof.

The optics block 118 magnifies received light from the electronicdisplay 115, corrects optical errors associated with the image light,and the corrected image light is presented to a user of the VR headset105. An optical element may be an aperture, a Fresnel lens, a convexlens, a concave lens, a filter, or any other suitable optical elementthat affects the image light emitted from the electronic display 115.Moreover, the optics block 118 may include combinations of differentoptical elements. In some embodiments, one or more of the opticalelements in the optics block 118 may have one or more coatings, such asanti-reflective coatings.

Magnification of the image light by the optics block 118 allows theelectronic display 115 to be physically smaller, weigh less, and consumeless power than larger displays. Additionally, magnification mayincrease a field of view of the displayed media. For example, the fieldof view of the displayed media is such that the displayed media ispresented using almost all (e.g., 110 degrees diagonal), and in somecases all, of the user's field of view. In some embodiments, the opticsblock 118 is designed so its effective focal length is larger than thespacing to the electronic display 115, which magnifies the image lightprojected by the electronic display 115. Additionally, in someembodiments, the amount of magnification may be adjusted by adding orremoving optical elements.

The optics block 118 may be designed to correct one or more types ofoptical error. Examples of optical error include: two dimensionaloptical errors, three dimensional optical errors, or some combinationthereof. Two dimensional errors are optical aberrations that occur intwo dimensions. Example types of two dimensional errors include: barreldistortion, pincushion distortion, longitudinal chromatic aberration,transverse chromatic aberration, or any other type of two-dimensionaloptical error. Three dimensional errors are optical errors that occur inthree dimensions. Example types of three dimensional errors includespherical aberration, comatic aberration, field curvature, astigmatism,or any other type of three-dimensional optical error. In someembodiments, content provided to the electronic display 115 for displayis pre-distorted, and the optics block 118 corrects the distortion whenit receives image light from the electronic display 115 generated basedon the content.

The locators 120 are objects located in specific positions on the VRheadset 105 relative to one another and relative to a specific referencepoint on the VR headset 105. A locator 120 may be a light emitting diode(LED), a corner cube reflector, a reflective marker, a type of lightsource that contrasts with an environment in which the VR headset 105operates, or some combination thereof. In embodiments where the locators120 are active (i.e., an LED or other type of light emitting device),the locators 120 may emit light in the visible band (˜380 nm to 750 nm),in the infrared (IR) band (˜750 nm to 1 mm), in the ultraviolet band (10nm to 380 nm), some other portion of the electromagnetic spectrum, orsome combination thereof.

In some embodiments, the locators 120 are located beneath an outersurface of the VR headset 105, which is transparent to the wavelengthsof light emitted or reflected by the locators 120 or is thin enough notto substantially attenuate the wavelengths of light emitted or reflectedby the locators 120. Additionally, in some embodiments, the outersurface or other portions of the VR headset 105 are opaque in thevisible band of wavelengths of light. Thus, the locators 120 may emitlight in the IR band under an outer surface that is transparent in theIR band but opaque in the visible band.

The IMU 130 is an electronic device that generates fast calibration databased on measurement signals received from one or more of the positionsensors 125. A position sensor 125 generates one or more measurementsignals in response to motion of the VR headset 105. Examples ofposition sensors 125 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, another suitable type of sensorthat detects motion, a type of sensor used for error correction of theIMU 130, or some combination thereof. The position sensors 125 may belocated external to the IMU 130, internal to the IMU 130, or somecombination thereof.

Based on the one or more measurement signals from one or more positionsensors 125, the IMU 130 generates fast calibration data indicating anestimated position of the VR headset 105 relative to an initial positionof the VR headset 105. For example, the position sensors 125 includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, roll). In some embodiments, the IMU 130rapidly samples the measurement signals and calculates the estimatedposition of the VR headset 105 from the sampled data. For example, theIMU 130 integrates the measurement signals received from theaccelerometers over time to estimate a velocity vector and integratesthe velocity vector over time to determine an estimated position of areference point on the VR headset 105. Alternatively, the IMU 130provides the sampled measurement signals to the VR console 110, whichdetermines the fast calibration data. The reference point is a pointthat may be used to describe the position of the VR headset 105. Whilethe reference point may generally be defined as a point in space;however, in practice the reference point is defined as a point withinthe VR headset 105 (e.g., a center of the IMU 130).

The IMU 130 receives one or more calibration parameters from the VRconsole 110. As further discussed below, the one or more calibrationparameters are used to maintain tracking of the VR headset 105. Based ona received calibration parameter, the IMU 130 may adjust one or more IMUparameters (e.g., sample rate). In some embodiments, certain calibrationparameters cause the IMU 130 to update an initial position of thereference point so it corresponds to a next calibrated position of thereference point. Updating the initial position of the reference point asthe next calibrated position of the reference point helps reduceaccumulated error associated with the determined estimated position. Theaccumulated error, also referred to as drift error, causes the estimatedposition of the reference point to “drift” away from the actual positionof the reference point over time.

The imaging device 135 generates slow calibration data in accordancewith calibration parameters received from the VR console 110. Slowcalibration data includes one or more images showing observed positionsof the locators 120 that are detectable by the imaging device 135. Theimaging device 135 may include one or more cameras, one or more videocameras, any other device capable of capturing images including one ormore of the locators 120, or some combination thereof. Additionally, theimaging device 135 may include one or more filters (e.g., used toincrease signal to noise ratio). The imaging device 135 is configured todetect light emitted or reflected from locators 120 in a field of viewof the imaging device 135. In embodiments where the locators 120 includepassive elements (e.g., a retroreflector), the imaging device 135 mayinclude a light source that illuminates some or all of the locators 120,which retro-reflect the light towards the light source in the imagingdevice 135. Slow calibration data is communicated from the imagingdevice 135 to the VR console 110, and the imaging device 135 receivesone or more calibration parameters from the VR console 110 to adjust oneor more imaging parameters (e.g., focal length, focus, frame rate, ISO,sensor temperature, shutter speed, aperture, etc.).

The VR input interface 140 is a device that allows a user to send actionrequests to the VR console 110. An action request is a request toperform a particular action. For example, an action request may be tostart or end an application or to perform a particular action within theapplication. The VR input interface 140 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the received action requests to the VR console 110. Anaction request received by the VR input interface 140 is communicated tothe VR console 110, which performs an action corresponding to the actionrequest. In some embodiments, the VR input interface 140 may providehaptic feedback to the user in accordance with instructions receivedfrom the VR console 110. For example, haptic feedback is provided whenan action request is received, or the VR console 110 communicatesinstructions to the VR input interface 140 causing the VR inputinterface 140 to generate haptic feedback when the VR console 110performs an action.

The VR console 110 provides media to the VR headset 105 for presentationto the user in accordance with information received from one or more of:the imaging device 135, the VR headset 105, and the VR input interface140. In the example shown in FIG. 1, the VR console 110 includes anapplication store 145, a tracking module 150, and a virtual reality (VR)engine 155. Some embodiments of the VR console 110 have differentmodules than those described in conjunction with FIG. 1. Similarly, thefunctions further described below may be distributed among components ofthe VR console 110 in a different manner than is described here.

The application store 145 stores one or more applications for executionby the VR console 110. An application is a group of instructions, thatwhen executed by a processor, generates content for presentation to theuser. Content generated by an application may be in response to inputsreceived from the user via movement of the HR headset 105 or the VRinterface device 140. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

The tracking module 150 calibrates the VR system 100 using one or morecalibration parameters and may adjust one or more calibration parametersto reduce error in determination of the position of the VR headset 105.For example, the tracking module 150 adjusts the focus of the imagingdevice 135 to obtain a more accurate position for observed locators onthe VR headset 105. Moreover, calibration performed by the trackingmodule 150 also accounts for information received from the IMU 130.Additionally, if tracking of the VR headset 105 is lost (e.g., theimaging device 135 loses line of sight of at least a threshold number ofthe locators 120), the tracking module 140 re-calibrates some or all ofthe system environment 100.

The tracking module 150 tracks movements of the VR headset 105 usingslow calibration information from the imaging device 135. The trackingmodule 150 determines positions of a reference point of the VR headset105 using observed locators from the slow calibration information and amodel of the VR headset 105. The tracking module 150 also determinespositions of a reference point of the VR headset 105 using positioninformation from the fast calibration information. Additionally, in someembodiments, the tracking module 150 may use portions of the fastcalibration information, the slow calibration information, or somecombination thereof, to predict a future location of the headset 105.The tracking module 150 provides the estimated or predicted futureposition of the VR headset 105 to the VR engine 155.

The VR engine 155 executes applications within the system environment100 and receives position information, acceleration information,velocity information, predicted future positions, or some combinationthereof of the VR headset 105 from the tracking module 150. Based on thereceived information, the VR engine 155 determines content to provide tothe VR headset 105 for presentation to the user. For example, if thereceived information indicates that the user has looked to the left, theVR engine 155 generates content for the VR headset 105 that mirrors theuser's movement in a virtual environment. Additionally, the VR engine155 performs an action within an application executing on the VR console110 in response to an action request received from the VR inputinterface 140 and provides feedback to the user that the action wasperformed. The provided feedback may be visual or audible feedback viathe VR headset 105 or haptic feedback via the VR input interface 140.

FIG. 2A is a diagram of a virtual reality (VR) headset, in accordancewith an embodiment. The VR headset 200 is an embodiment of the VRheadset 105, and includes a front rigid body 205 and a band 210. Thefront rigid body 205 includes an electronic display 115, the IMU 130,the one or more position sensors 125, and the locators 120. In theembodiment shown by FIG. 2A, the position sensors 125 are located withinthe IMU 130, and neither the IMU 130 nor the position sensors 125 arevisible to the user.

The locators 120 are located in fixed positions on the front rigid body205 relative to one another and relative to a reference point 215. Inthe example of FIG. 2A, the reference point 215 is located at the centerof the IMU 130. Each of the locators 120 emit light that is detectableby the imaging device 135. Locators 120, or portions of locators 120,are located on a front side 220A, a top side 220B, a bottom side 220C, aright side 220D, and a left side 220E of the front rigid body 205 in theexample of FIG. 2A.

FIG. 2B is a cross section 225 of the front rigid body 205 of theembodiment of a VR headset 200 shown in FIG. 2A. As shown in FIG. 2B,the front rigid body 205 includes an optical block 230 that providesaltered image light to an exit pupil 250. The exit pupil 250 is thelocation of the front rigid body 205 where a user's eye 245 ispositioned. For purposes of illustration, FIG. 2B shows a cross section225 associated with a single eye 245, but another optical block,separate from the optical block 230, provides altered image light toanother eye of the user.

The optical block 230 includes an electronic display 115, and the opticsblock 118. The electronic display 115 emits image light toward theoptics block 118. The optics block 118 magnifies the image light, and insome embodiments, also corrects for one or more additional opticalerrors (e.g., distortion, astigmatism, etc.). The optics block 118directs the image light to the exit pupil 250 for presentation to theuser.

FIG. 3A is a top view and FIG. 3B is a cross section of an electronicdisplay 115, in accordance with an embodiment. In one embodiment, theelectronic display 115 is a LCD device including a LC panel 310, BLU320, a data driver 330, and a controller 340. The LC panel 310 coversthe BLU 320 and includes a pixel area 302 comprising a plurality of rowsof pixels including a first row 304 and a last row 306 of pixels. Across section of the pixel area 302 along line 312 is shown in FIG. 3Band shows the LC panel 310 covering the BLU 320.

The BLU 320 includes a light source (not shown) that is an electricalcomponent that generates light. The BLU 320 is configured to emit lighttoward the LC panel 310. The light source may comprise a plurality oflight emitting components (e.g., light emitting diodes (LEDs), lightbulbs, or other components for emitting light). In one aspect, intensityof light from the light source is adjusted according to a backlightcontrol signal from the controller 340. The backlight control signal isa signal indicative of intensity of light to be output for the lightsource. A light source may adjust its duty cycle of or an amount ofcurrent supplied to the light emitting component (e.g., LED), accordingto the backlight control signal. For example, the light source may be‘ON’ for a portion of a frame time, and ‘OFF’ for another portion of theframe time, according to the backlight control signal. Exampleoperations of the BLU 320 are further described in detail below withrespect to FIGS. 4A to 4C. The BLU 320 projects light from the lightsource towards the LC panel 310. The BLU 320 may include a light guideplate and refractive and/or reflective components for projecting lighttowards the LC panel 310. The light guide plate may receive light withdifferent colors from light sources, and may project combined lightincluding a combination of the different colors towards the LC panel310.

The LC panel 310 includes a bottom substrate 322, a top substrate 324,and LC material 326 between the bottom and top substrates 322 and 324.Although not shown in FIG. 3B, the bottom substrate 322 may includedriver pixel circuitry and transparent pixel electrodes, and the topsubstrate 324 may include color filters, a black matrix, and transparentconductive electrodes. Also, spacers may be used to control the spacingbetween the top substrate and the bottom substrate, although not shownin FIG. 3B. The LC material 326 is placed between the top and bottomsubstrate 322 and 324.

The data driver 330 is coupled to the LC panel 310 and writes displaydata (e.g., data corresponding to an image) to pixels in the pixel area302 of the LC panel 310. Although shown as a separate component, thedata driver 330 may be included in the LC panel 310. The data driver 330writes the display data in a scan direction 314 from a first row 304 toa last row 306 of pixels in the pixel area 302. The display data writtento a pixel may be in the form of an analog voltage that may be appliedacross electrodes on the bottom and/or top substrate 322 and 324 of apixel to change the orientation of LC material 326 in the LC panel 310.The change in orientation of the LC material 326 allows a portion of thelight from the BLU 320 to reach a user's eye 245.

The controller 340 is a circuit component that receives an input imagedata and generates control signals for driving the data driver 330 andBLU 320. The input image data may correspond to an image or a frame of avideo in a VR and/or AR application. The controller 340 instructs thedata driver 330 to write data to the LC panel 310 to control an amountof light from the BLU 320 to the exit pupil 250 through the LC material326. The controller 340 generates the backlight control signal forturning ON or OFF the BLU 320, as described in more detail for FIG. 4A.In other embodiments, the electronic display 115 includes different,more or fewer components than shown in FIGS. 3A and 3B. For example, theelectronic display 115 may include a polarizer and a light diffusingcomponent.

Black Duty Insertion Mode for LCDs in VR Headset

The electronic display 115 in a VR headset has certain requirements suchas a short duty cycle to prevent image streaking and short illuminationtimes to reduce latency. While the electronic display 115 could be aLiquid Crystal Display (LCD), LCDs are currently one or two orders ofmagnitude slower than active matrix OLED displays (AMOLEDs). Theswitching time associated with the liquid crystal (LC), or the amount oftime required for the LC to change state, may take several milliseconds(ms), making it difficult to achieve a short duty cycle with LCDs andlimiting the speed of LCDs. In addition, conventional LCD has thebacklight unit (BLU) always turned on and do not have short illuminationtimes. To improve LCD performance in a VR headset, a shorter duty cycleand illumination time may be achieved by using an alternative operatingmode for LCDs such as a BDI mode according to the embodiments herein.

In the BDI mode, the entire frame time is made available for image datascan out (and charging of the pixels of the LCD accordingly) and LCstate change according to the image data. In one aspect, the data driverwrites data to all of the pixels of the LCD before the BLU emits light.The BLU is typically turned on for a portion (e.g., 20%) of the frametime, for example during the last 20% of the frame. In this mode, theBLU may turn on while the image data for some of the pixels is stillbeing scanned out for charging into the pixels or while the LC of someof the pixels is still undergoing transition according to the chargedimage data. The resulting image that is shown during the illuminationportion of the BLU may include compromised pixels which have notcompleted the LC transition to the state indicated by the written imagedata, and old images of pixels from a previous frame which are beingupdated during the illumination portion of the BLU. By turning on theBLU during the final portion of the frame—as opposed to the beginning ormiddle portion of the frame—the number of compromised pixel may bereduced.

FIGS. 4A-4C show an example frame time for a 90 Hz LCD in BDI mode,according to one embodiment. In the example frame time shown in FIG. 4A,the allocated period for data scan out is represented from t_(s) tot_(e), where t_(s) represents start of the frame and t_(e) representsthe end of the frame. The illumination of the BLU begins at t_(i)irrespective of whether the pixel data of the whole frame is scanned outand their corresponding LCs have transitioned, and ends at the end ofthe frame, t_(e). For an example frame time of 11 ms in the BDI mode,data scan out and LC state change are allocated the entire 11 ms and theillumination time of the BLU is set to the final 2 ms of the frame time.During the illumination portion of the BLU, pixels updated during abeginning portion of the frame time (e.g., the first 3 ms) displays datathat is updated and correct; pixels updated during a middle portion ofthe frame time (e.g., from 3 ms to 9 ms) may be in a compromised state,and pixels updated during a last portion of the frame time (e.g., thelast 2 ms) may display old images from a previous frame.

In an LCD running with the BDI mode where the pixels are updated from atop row to a bottom row, the bottom rows of the LCD may displaycompromised or old image data. For example, FIG. 4B shows the timingdiagram corresponding to the first row of pixels of the LCD. As shown inFIG. 4B, image data for the first row is scanned out at the beginning ofthe frame time (i.e., closer to time t_(s)). The LCs corresponding tothe first row of pixels begin changing their state as soon as the datascan out is complete and finish their transition earlier than the end ofthe allocated LC state change time budget (i.e., earlier than timet_(e)). FIG. 4C shows the timing diagram corresponding to the last rowof pixels of the LCD. As shown in FIG. 4C, image data for the last rowis scanned out later in the frame time such that the LCs correspondingto the last row of pixels finish their transition closer to the end ofthe allocated LC state change time budget (i.e., closer to time t_(e)).As shown in FIG. 4B, the BLU illumination begins before all LCscorresponding to the last row of pixels have completed their transitions(i.e., at time t_(i)) thereby resulting in some compromised pixels. Thedata scan out and LC state change for all other rows of pixels occurs attimes that are in between the use cases described for the first row(i.e., FIG. 4B) and the last row (i.e., FIG. 4C). Embodiments of the BDImode are further described in U.S. Provisional Application No.62/325,920, filed on Apr. 21, 2016, which is hereby incorporated byreference in its entirety.

Eye Tracking

FIG. 5 shows a cross section of a headset e.g., headset 100 or a HMD insome other system (e.g., an AR system) including a camera for trackingeye position, in accordance with at least one embodiment. The headsetincludes two separate cameras 502, one camera for tracking the positionof each eye 500 and may include an eye tracking module 508 (e.g., eyetracking device). The eye tracking module 508 may be coupled to the twoseparate cameras 502 and the electronic display 115. The eye trackingmodule 508 may receive image data from the two separate cameras 502 andmay provide information (e.g., location of an eye gaze area orillumination start time for a BLU) to the electronic display 115. In theexample shown in FIG. 5, cameras 502 capture images of the user's eyesand an eye tracking module 508 within the headset 100 determines gazelines 504 corresponding to a location where the user is looking based onthe captured images. Here, the user is looking at object 506, which maybe a real object (e.g., in AR applications) or a virtual object (e.g.,in VR applications).

In one embodiment, the eye tracking module 508 performs eye tracking byfocusing a camera on one or both of the user's eyes and records theirmovements as the user looks at some kind of stimulus. The stimulus canbe light sources that emit light in either infrared, near-infrared,visible light, or some combination thereof. The eye tracking module 508tracks the center of the eye's pupil by capturing eye images at a rateof, for example, 60-300 times per second, and inferring the pupillocation from the images using a computer algorithm. The eye trackingmodule 508 estimates an intersection of gaze lines 404 corresponding toeach of the user's eyes by extrapolating the gaze lines 504 until theyboth intersect. The eye tracking module 508 may use any of the knowntechniques in the art for determining the user's gaze direction for eachof the user's eyes. Embodiments of the eye tracking module 508 arefurther described in U.S. patent application Ser. No. 14/946,143, filedon Nov. 19, 2015, and U.S. Provisional Patent Application No.62/306,758, filed on Mar. 11, 2016, which are hereby incorporated byreference in its entirety.

Backlight Modulation Using Eye Tracking

FIGS. 6A-6C show diagrams illustrating BLU illumination for displayingimage data on an LCD and their corresponding timing diagrams, inaccordance with an embodiment. The device system (e.g., systemenvironment 100) utilizes eye tracking to determine a user's gaze whilethe user is viewing displayed image data on LCD as described above withreference to FIG. 5. The embodiment depicted in FIGS. 6A-6C displaysimage data using the BDI mode. As described above with reference to FIG.4A-4C, the BLU is turned on for only a portion of the frame time in theBDI mode to display the entire pixels of the LCD. Conventionally, theBLU is turned on during the final portion of the frame, for exampleduring the last 20% of the frame, and pixels corresponding to the bottomhalf of the LCD may either be in a compromised state or may display oldimage data from a previous frame. By using eye tracking, the BLU may beturned on at a time when the LCs corresponding to the pixels in theuser's gaze of the current frame (e.g., frame n) have fully transitionedto their final state. Such modulation of the BLU using eye tracking mayimprove user experience by reducing the number of pixels either in acompromised state or displaying old image data from a previous frame.

FIG. 6A depicts an example use case where the user is looking at the topportion of the LCD. The top portion of FIG. 6A shows pixel area 650 ofthe LCD and the bottom portion shows a timing diagram 655. Pixel area650 indicates that the user is currently looking at the top portion ofthe LCD represented by eye gaze area 602. The timing diagram 655 showsthe period for data scan out and LC state change represented from t_(sn)to t_(en), where t_(sn) represents the start time of the frame n andt_(en) represents the end time of the frame n. The BLU illuminationbegins at tin′ and ends at t_(ien1). In one embodiment, t_(ien1) mayoccur at the end time of frame n. The illumination of the BLU is settowards the last portion of the frame time of frame n such that the LCscorresponding to the pixels the user is looking at (i.e., eye gaze area602) have completed their transition to their final state by the timethe BLU illumination begins at t_(in1). Pixel area 650 shows region 604that corresponds to the portion of the LCD whose corresponding LCs mightnot have completed their transitioning at the time t_(in1). When BLU isilluminated within frame n, some pixels of the region 604 show imagedata corresponding to the frame n−1 that is immediately prior to frame nand the other pixels show image data corresponding to the current framen. However, this may not have adverse effect on the user experiencebecause the eye gaze area is toward region 602 and not region 604.

FIG. 6B depicts an example use case where the user is looking at themiddle portion of the LCD. Pixel area 660 indicates that the user iscurrently looking at the middle portion of the LCD represented by eyegaze area 612. The timing diagram 665 shows the period for data scan outand LC state change represented from t_(sn) to t_(en) for frame n andfrom t_(sn+1) to t_(en+1) for the immediately next frame n+1. Theillumination of the BLU is set towards the last portion of the frametime of frame n and the first portion of frame time of frame n+1 suchthat the LCs corresponding to the pixels of the middle portion of theLCD (i.e., eye gaze area 612 that the user is looking at) have completedtheir transition to their final state by the time the BLU illuminationbegins at t_(in2). The BLU illumination begins at t_(in2) and ends att_(ien2), where t_(ien2) is a time within the frame time of theimmediately next frame, n+1. In comparison to the example shown in FIG.6A, t_(in2) occurs later in the frame time than tin′. The BLUillumination time period (i.e., from t_(in2) to t_(ien2)) may be variedbased on the tracking the user eye gaze. For example, if the user's gazemoves a few rows below from the gaze shown in FIG. 6B, the BLUillumination time period is shifted to the right (i.e., delayed) toaccount for the change in the user's gaze. On the other hand, if theuser's gaze moves a few rows above from the gaze shown in FIG. 6B, theBLU illumination time period is shifted to the left (i.e., moved tooccur earlier) to account for the change in the user's gaze. Pixel area660 shows regions 614 and 616 that corresponds to the portions of theLCD whose corresponding LCs might not have completed their statetransition at the time t_(in2). When BLU is illuminated within frame n,some pixels of the region 614 show image data corresponding to the framen−1 that is immediately prior to frame n and some pixels of the region616 show image data corresponding to the frame n+1 that is immediatelyafter frame n. However, this may not have adverse effect on the userexperience because the eye gaze area is toward region 612 and notregions 614, 616.

FIG. 6C depicts an example use case where the user is looking at thebottom portion of the LCD. Pixel area 670 indicates that the user iscurrently looking at the bottom portion of the LCD represented by eyegaze area 622. The timing diagram 675 shows the period for data scan outand LC state change represented from t_(sn) to t_(en) for frame n andfrom t_(sn+1) to t_(en+1) for the immediately next frame n+1. The BLUillumination begins at t_(in3) and ends at t_(ien3). In comparison tothe example shown in FIG. 6B, t_(in3) occurs later in the frame timethan t_(in2). The illumination of the BLU for frame n is set within thebeginning portion of the frame time corresponding to frame n+1 such thatthe LCs corresponding to the pixels of the bottom portion of the LCD(i.e., eye gaze area 622 that the user is looking at) have completedtheir transition to their final state by the time the BLU illuminationbegins at t_(in3). Pixel area 670 shows region 624 that corresponds tothe top portion of the LCD whose corresponding LCs might not havecompleted their transitioning at the time t_(in3). When BLU isilluminated within frame n, some pixels of the region 624 show imagedata corresponding to the frame n+1 that is immediately next to frame nand the other pixels show image data corresponding to the current framen. However, this may not have adverse effect on the user experiencebecause the eye gaze area is toward region 622 and not region 624.

While modulating BLU based on the user's eye tracking in the BDI modemight result in some compromised pixels or some amount of old image datacorresponding to immediately prior frame n−1 (or new data correspondingto immediately next frame n+1), some amount of such compromised imagedata is acceptable as human eyes might not perceive such difference. Inaddition, BLU modulation ensures that the portion of the LCD the user isviewing corresponds to the portion whose LCs have more fullytransitioned to their final states and thereby confining the portion ofthe LCD displaying compromised image data to the user's peripheralvision. In some embodiments, the VR Engine 155 may make slightadjustments in the latency correction based on the BLU timing changesrelative to the frame.

Additional Configuration Information

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the patent rights be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A head mounted display system comprising: adisplay device including: a liquid crystal (LC) panel comprising aplurality of rows of pixels; a back light unit (BLU) configured to emitlight toward the LC panel; a data driver configured to write datacorresponding to an image to the LC panel; and a controller configuredto apply a first control signal to the data driver to write the data tothe LC panel and a second control signal to the BLU to emit light duringan illumination period of a frame period from an illumination start timeand to not emit light for a remaining portion of the frame period; andan eyetracking device configured to determine an eye gaze area of a userin a pixel area of the display device, wherein the illumination starttime varies based on a location of the eye gaze area such that theillumination start time is offset from a beginning of the frame periodby a first amount and another illumination start time is offset from abeginning of another frame period by a second amount that varies, fromthe first amount, based on the location of the eye gaze area.
 2. Thehead mounted display system of claim 1, wherein liquid crystal materialin a row of the pixels of the LC panel outside the eye gaze area of theuser transitions during the illumination period.
 3. The head mounteddisplay system of claim 1, wherein when the eye gaze area of the user isin a top portion of the pixel area, the illumination start time occursbefore an end of the frame period.
 4. The head mounted display system ofclaim 1, wherein when the eye gaze area of the user is in a bottomportion of the pixel area of the display device, the illumination starttime is at an end of the frame period or a beginning of a subsequentframe period.
 5. The head mounted display system of claim 1, wherein thedata driver writes the data to a row of the pixels of the LC paneloutside the eye gaze area in a subsequent frame period during at least aportion of the illumination period.
 6. The head mounted display systemof claim 1, wherein when the eye gaze area of the user is in a middleportion of the pixel area of the display device, the illumination starttime occurs before an end of the frame period and the illuminationperiod extends into a subsequent frame period subsequent to said frameperiod.
 7. The head mounted display system of claim 6, whereinresponsive to the eye gaze area of the user moving below a previous eyegaze area of the user, the illumination start time shifts closer to anend of the frame period.
 8. The head mounted display system of claim 6,wherein responsive to the eye gaze area of the user moving above aprevious eye gaze area of the user, the illumination start time shiftscloser to a start of the frame period.
 9. The head mounted displaysystem of claim 1, wherein the data driver writes the data to theplurality of the rows of the pixels at a beginning portion of the frameperiod.
 10. The head mounted display system of claim 9, wherein theillumination start time occurs at or after an end of the beginningportion of the frame period.
 11. A method of displaying an image by adisplay device, the method comprising: writing, by a data driver, datacorresponding to the image to a liquid crystal (LC) panel comprising aplurality of rows of pixels; determining, by an eyetracking device, aneye gaze area of a user in a pixel area of the display device; emitting,by a back light unit, light toward the LC panel during an illuminationperiod of a frame from an illumination start time and not emitting lightfor a remaining portion of the frame period, wherein the illuminationstart time varies based on a location of the eye gaze area such that theillumination start time is offset from a beginning of the frame periodby a first amount and another illumination start time is offset from abeginning of another frame period by a second amount that varies, fromthe first amount, based on the location of the eye gaze area.
 12. Themethod of claim 11, wherein liquid crystal material in a row of thepixels of the LC panel outside the eye gaze area of the user transitionsduring the illumination period.
 13. The method of claim 11, wherein whenthe eye gaze area of the user is in a top portion of the pixel area, theillumination start time occurs before an end of the frame period. 14.The method of claim 11, wherein when the eye gaze area of the user is ina bottom portion of the pixel area of the display device, theillumination start time is at an end of the frame period or a beginningof a subsequent frame period.
 15. The method of claim 11, furthercomprising: writing, by the data driver, the data to a row of the pixelsof the LC panel outside the eye gaze area in a subsequent frame periodduring at least a portion of the illumination period.
 16. The method ofclaim 11, wherein when the eye gaze area of the user is in a middleportion of the pixel area of the display device, the illumination starttime occurs before an end of the frame period such that the illuminationperiod extends into a subsequent frame period subsequent to said frameperiod.
 17. The method of claim 16, wherein responsive to the eye gazearea of the user moving below a previous eye gaze area of the user, theillumination start time shifts closer to an end of the frame period. 18.The method of claim 16, wherein responsive to the eye gaze area of theuser moving above a previous eye gaze area of the user, the illuminationstart time shifts closer to a start of the frame period.
 19. The methodof claim 11, wherein the data driver writes the data to the plurality ofthe rows of the pixels at a beginning portion of the frame period. 20.The method of claim 19, wherein the illumination start time occurs at orafter an end of the beginning portion of the frame period.